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ABSTRACT 

The use of embedded support devices (ESDs) has been 
studied in previous rese.irch that lias unfortunately demanded 
introspection and reflection on one's own cognitive functioning. In 
contrast, eye movement registration measures the attentional focus of 
students on specific text oarts regardless of whether or not the 
student is conscious of this focus. Earlier research has indicated 
that ESDs are used at a deep level, and not superficially read. 
Eleven college students read slides from a chapter of a governmental 
law course from the Open University of the Netherlands. Eye movements 
were monitored through the corneal reflection/pupil center technique 
and computer analysis. Eye movement patterns were compared for text 
with ESDs and text without ESDs. The average reading time per word 
was in line with expectations for most ESDs. Every ESD was looked p-t 
least once by each student. Limitations and possibilities for 
improvement of the research technique are discussed. Five figures and 
six tables illustrate study findings. (Contains 67 references.) 
(SLD) 



>'t i< i: Vt Vc Vc i< >'t Vt i'< I'l i'< it it it it it it ■' it it it it it it it it it it it it it it it it i' it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it >' 

RcpruducL ! one Gupplicd by EDRS arc the bent thai can be made 

from the original document. ' 

i: it it it it it it it it it it it it it it it it it it it it i: it it it it it it it it it it it it it it it it it it it it it it it it it it i: it it i: it it it it it it it i: it it it it it it it it k it i 



U.S. OEPARTMENT OF EDUCATION 
0tfiC9 of Educaltonol Resaarch and Improvemenl 

EDUCATIONAL RESOURCES INFORMATION 
K CENTER (ERIC) 

rJOhit doeumeni his been lopioduced ns 
Trocoivod from the peraon or OfQanization 

originating il 
□ Minor changes have been made lo improve 
reproduction quality 

« Pointa ol view or opinionsstflied in this docu- 
ment do not necessarily represent otiiciai 
OERl position or policy 



Eye ■ movement re, 



-PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 




use 



r- 

m 
oo 
cn 

Q 



R.L. Martens 
P.H.A,G. Foelmans 



1 
ER|C 



COPY AVAILABLE 



2 



OTIC RESEARCH REPORTS. 



The Open university is responsible for devefoping and offering open, higher distance education in 
which special attention is paid to innovation; in educational technology. The research in this field 
is concentrated in OTIC, that is the Centre for Educational Technology and Innovation 
(Onderwijstechnologisch Innovatie Centrum). 

OTIC is also engaged in running projects for other institutes. Here the Centre makes use of OTIC's 
knowledge and experience acquired in research and development. 

The series of OTIC Research Reports consists of publications of the OTIC research projects and 
aims mainly at an audience of fellow researchers. 



RESEARCH PROJECT Course versions 

The project Interactive learning and course development environment (ILCE) focuses on the 
construction and validation of -components of- an interactive learning and course development 
environment that enable flexibility of course development and stud)dng. Interactive learning 
environments nuike it possible to adapt course materials to individual s*aident needs and 
characteristics such as prior knowledge. An important input for this project is the project 'course 
versions' (cursusvarianten) that focused on the effects of different types of course variants 
(versions) in printed distance education material as an important approach towards student 
support. 

The main objectives are in the field of flexible organization of support, incorjporation of support 
into learning materials, optimizing the link between student characteristics and the support needs, 
and the development of more flexible (and automatised) approaches towards student guidance. 
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Eye movement registration during studying printed self study material: the use of embedded support devices 



Abstract 



In previous OTlC-research the use of embedded support devices (ESD) has been investigated by 
making use of interviews and questionnaires. An important disadvantage of these research 
methods is that they demand introspection and reflection on one's own cognitive functioning. In 
contrast, eye movement registration is very different in this perspective. The attentional focus of 
students on specific text parts is registered, regardless of the question whether the student is 
conscious about tliis. Using a research set-up based on this method we wanted to check and 
replicate findings of our earlier research on the use of ESD in self-instructional materials: ESD are 
used at a deep level and are not superficially read. 

Eleven subjects paiticipated in this research. Slides were projected with a projector on a screen in 
front of the subjects. The slides depicted pages of a learning unit (chapter) of the Ou-course 
'Introduction to governmental law'. The eye movements were monitored by means of the corneal 
reflection/pupil centre technique with an infrared camera. With the computer programs 
'EYECATCH' and 'EYEANAL" the data of the eye movements were recorded and analyzed. Eye 
movement patterns were compared between text parts with ESD and those without ESD. For each 
text part the computer program computed the total reading time per selected area. From this the 
reading time per word was computed. In relation to 'schemes', also the number of 'jumps' between 
the text and back to the scheme were recorded. 

It was found that the average reading time per word was for most ESD in line with our 
expectations. For instance content pages have shorter reading times per word and questions are 
read longer per word. 'Schemes' invoke ± 3.6 jumps to other text parts. Furthermore it turned out 
that every ESD analyzed was looked at, at least once by each student. This first study, making use 
of the eye registration technique is to be considered as a first attempt to approach the study of ESD 
use with new research techniques. Shortcomings and possibilities for improvement are discussed. 
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1 Introduction 



In the context of distance education the quality of study materials is of prime importance. The 
study materials of the Open university of the Netherlands (Ou) are still mainly based on printed 
text. Much energy and time is invested to generate these materials. An essential part of the printed 
study text (up to 40%) consists of embedded support devices (e.g. content pages, examples, 
advanced organizers, pre- and postquestions, schemes, tests, margin texts). Embedded support 
devices (ESD) are defined as formal and /or content-related add-ons, extensions and elaborations of 
the learning content in printed study materials. They aim at scaffolding the learning process by 
supporting (a) the orientation, (b) the planning, (c) the processing and the (d) evaluative aspects of 
the individual learning process. 

This research has been set up as part of the project 'Course Versions' The project focuses on the 
effects of course versions on study outcomes. Especially, the effect of varying the amount of ESD is 
of main interest. The key question of this project is: "Is the high investment in the design and 
elaboration of printed study materials wortii the effort?" The key questions of the 'Course 
Versions' project are related to the 'effects' of ESD on study outcomes/results. The present research 
elaborates on findings of earlier OTIC-research by making use of a different research method. 

Research methods used in earlier research (for instance Martens, Poelmans, Daal, van Staa & van 
Meurs, 1993; Valcke, Martens, Poelmans, Daal, 1993) were based on questionnaires and interviews. 
An important disadvantage of these two research methods is that they build on introspection and 
reflection on one's own cognitive functioning. This is in sharp contrast with the overall accepted 
statement that a great deal of the study process occurs at subconscious level (Resnick, 1987). Using 
interviews or printed questionnaires therefore incorporates the risk of underestimating or 
mistaking functions and effects of ESD. Just & Carpenter (1977, p.l) state: "For most adults, 
comprehension is rapid, automatic and effortless. But, despite of its apparent simplicity, 
comprehension includes a myriad of subprocesses, each of which by itself constitutes a formidable 
computational task. (...) All these processes occur so quickly that their richness is underestimated. 
Because of the sheer speed of comprehension, it is difficult for the reader to explain how he came 
to understand a passage. That task is left to those who study comprehension." 
It is as a consequence difficult to get access to these processes, certainly if this is done by means of 
interviews or questionnaires. To overcome this problem other research methods are desirable 
(Valcke & Martens, 1992). The eye movement registration method is in this perspective very 
efficient since the research method does not build on reflection or monitoring of the personal study 
process. The attentional focus of students on specific text parts is registered, whether the student is 
conscious about this process or not. In this study, we will use eye movement registration 
techniques to investigate the use of ESD in course materials of the Ou. 

This research report starts with an overview of earlier research. Chapter 3 includes a short 
summary of the theoretical base and background of our research in order to state the research 
hypotheses. Chapter 4 focuses on the description of the research design. Attention is paid to the 
selection of the research sample, the research instruments, the eye movement registration 
procedure, the research procedure and the preparation of the data set for further analysis. 
Furthermore we pay attention to the potential distracting nature of the particular research design 
and reliability and validity issues. After discussing characteristics of the research sample, the 
analysis procedure of the research data is presented in chapter 5. In chapter 6 we discuss the 
research results in view of the hypotheses put forward. 
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2 Earlier research 



Embedded support devices, such as objectives, questions, schemes, examples, etc. have been the 
subject of a vast body of research since printed texts play a major role in educational contexts. In 
the seventies and eighties some major reviews of this research have been elaborated (cf. 
Alesandrini, 1984; Barnes & Clawson, 1975; Hartley & Davies, 1976; MacDonald-Ross, 1977; 
Rothkopf, 1970; Waller, 1979; etcetera). 

This section focuses firpt of all on earlier research conducted outside the Ou oi the 
Netherlands. In an earlier research report an extensive overview has already been presented 
(Valcke & Martens, 1992). therefore, we centre in this report on the general results of this overview 
and especially exprase important shortcomings of the research tradition in this field. 

In a second subpart an overview of research conducted at the Ou of the Netherlands is 
presented. This section shows how the research reported here builds on earlier findings and 
focuses on alternative research methods. 



2.1 Research 

Limited amount of research in a distance education context 

Clyde and Crowther (1983, p.4) clearly p^';nt at a major shortcoming of research in relation to 
distance education when they state: "Despite the growth of distance education, little systematic 
knowledge is available concerning the ways in which students use the distance teaching materials 
provided to them." (cf. also Marland, Patching, Putt & Putt, 1990). Despite the high investment in 
ESD by course developers, little is known about there functions and effects in study materials. This 
is particularly true if we focus on research in a 'distance education' context. 

Doubts about functions and effects of esd 

Most ESD are designed to evoke some kind of behaviour of students. Wade and Trathen (1989, p. 
40) however state: "(...) despite their popularity, there is no consistent empirical evidence that these 
techniques are any more effective than more passive methods of reading-only or repetitive 
reading...". Even when - from a theoretical point of view - functions and effects are assigned to 
specific ESD, there is t?\e problem that Meyer and Watson (1991) have signalled: practical 
implications derived from this are often not useful. 

Doubts about student use of esd 

MaTiy research on ESD is marred by the fact that researchers hardly know what learners do when 
they are presented with text-embedded support. It is for instance not sure whetiier or not they 
actually use or know how to use these support facilities (Winne, 1983). We must not forget that the 
student - when working at a distance with printed learning materials - has complete veto power 
over learning (Rothkopf, 1970). Winne (1983. p.245) states for example when he tries to explain the 
inconsistencies in earlier research results : "these results may reflect different ways that learners 
cognitively responded to the instructional stimuli rather than weaknesses in their effects". In order 
to obtain more valid results., this researcher trains his students to use the embedded support 
devices (objectives, adjunct postquestions). A similar approach is adopted by Bernard (1990) and 
de Jong & Simons (1988). 

The question is not only whether students use the support devices or whether they know how to 
use them, but also whether they need the support provided. ESD are only useful if students need 
them. Waller (1979) mentions e.g. different user objectives when dealing with study texts: 
entertainment, recapitulation /reviewing, browsing, studying in depth, searching for a particular 
item. Each user perspective affects the potential effects /functions of ESD. 
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Methodological problems 

Valcke & Martens (1992) stated that the ecological validity of many research set-ups is 
questionable. Research is hardly set up in a distance education context and mostly not in natural 
learning contexts. Most research is set up in isolated lab-like settings (cf. Lockwood, 1993). 

Theoretical problems 

Finally, serious theoretical problems can be put forward when analyzing the research in the 
literature. Valcke & Martens (1992) signalled a lack of theoretical base, reductionist approaches and 
behaviouristic influences that lead to the omission of potentially important variables and an 
unidimensional approach in which individual effects and interactions are neglected. 



2.2 Research at the Ou of the Netherlands 

At the Ou of the Netheriands, research in relation to ESD was set up since 1991. We describe in 
short some of the research-approaches and the main results. 

Theoretical framework 

A basic phase in the Ou-research was the elaboration of a theory on the potential functions and 
effects of embedded support devices in printed study materials (Valcke & Martens, 1992). This 
theoretical model is considered as a starting point from where to look at the actual use of ESD by 
students when they study course materials. 

In reviewing the literature, also methods and instruments were examined to investigate the use, 
functions and effects of embedded support devices. 



ChBcklist 

A checklist was developed to map the occurrence and quality of ESD in a study text, at two levels: 
course level and learning unit level (Valcke, Daal, Martens & Dochy, 1991). 
This checklist was used in a try-out vdth the course Introduction to Governmental Law' and 
proved to be useful for a systematic mapping of BSD in printed course materials (Martens & Daal, 
1992; Martens, Daal & Valcke, 1991). About 40% of the text of Ou study materials consists of ESD. 

Research vnth interoiew method 

In a first research set-up twenty-five law students of the Ou were interviewed about the way they 
study Ou-leaming materials. The interview gradually focused on the use and effects of ESD as 
perceived by the students (Poelmans, Martens, Daal, Valcke & Dochy, 1992; Valcke, Martens, 
Poelmans, Daal, 1993). The interview method had the advantage to reveal a considerable amount 
of detailed information. 25 law students were interviewed. 

The research results indicated that students use ESD to a high extent (varying from 60% to 100%) 
and highly appreciate them. Students who use ESD to a high extent at a deep level, attain a 
significantly higher final test score and need less attempts to pass the final test (Poelmans et al, 

1992) . 'Deep level' refers to a concept commonly found in literature about study regulation: surface 
level versus deep level studying (Marton & Saljo, 1984). Studying at a surface level means 'only' 
reading and learning by heart, deep level learning puts emphasis on comprehension. 

Research xoith questionnaire method 

At the Ou of the Netheriands, the department OID sets up - yeariy - an annual evaluative study 
involving about 2000 students. This annual research is based on the use of questionnaires. Students 
taking courses in different subject domains participate in the study. In 1992, a part of the 
quesHonnaire consisted of quesHons about ESD their use and links with study success. Researchers 
of OID and OTIC analyzed the research data (Martens, Poelmans, Daal, van Staa & van Meurs, 

1993) . The results revealed that students appreciate ESD positively: only 7% of the students 
appreciate course material without ESD. Students of the subject domain 'Cultural Sciences' 
appreciated ESD to a lesser extent (Pearson correlation coefficient -.24, p<0,0001). No interrelations 
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between the use of ESD and the level of study success could be found. Study success was in this 
context operationalised as the speed to attain study credits. For more information about this 
research with questionnaires and problems that were encountered we refer to van Staa & van 
Meurs (1992). 

Different versions of one course 

In another research, the relation between ESD and study results was investigated by adopting a 
different research design and method. The key question the research focused on was: do ESD have 
a positive effect on study outcomes? Two versions of a learning unit of the Ou-course 
'Introduction to governmental law' were assembled. The first version was almost an exact copy of 
the original printed text in the Ou-textbook. The second version was about 60% shorter as a result 
of deleting most embedded support devices (such as schemes, summaries and learning objectives). 
This omission did not alter the basic content or did not affect the coherence of the text. Thirty-six 
students from the University of Limburg (RUL) studied one of both course versions. Several tests 
and questionnaires were developed to measure the effects of course versions on study success and 
the p..tential interrelation of study success with independent student variables. No significant 
effects on study time, student evaluations of flie learning access level and 'knowledge' questions 
could be observed. But students who studied the extensive course version, comprising ESD, 
achieved significant higher score? on 'insighf questions (T-test with F=3.47, P<0.05). Interrelations 
with the independent variables prior knowledge, time used, reading comprehension, educationa. 
level and gender were tested. These variables were added to the ANOVA models as covanables. 
Reading comprehension turned out to have a significant effect (Martens, Poelmans, Daal, Valcke, 
1993). 



3 Theoretical base 



Much effort was invested in developing a theoretical framework to describe and explain the 
functions and effects of embedded support devices in relation to studying from pnnted texts. This 
section summarizes the theoretical base of this research. 

First of all key processes and variables are presented. The following variables and processes are 
considered to be of importance: cognitive aptitudes (3.1 cognitive functioning and 3.2 reading 
comprehension), non-cognitive aptitudes (3.5 motivation), task environment (3.3) and intra- 
individual variables (3.4). In section 3.6 we integrate these processes anH variables >nto one 
consistent theoretical model grounding this research. We refer to Valcke & Martens (1992) for a 
more extensive description and discussion of the model. 

A short overview of eye movement registi-ation research is given in section 3.7. This speaal 
research method has to be related to our research set-up and theoretical model. 
Section 3.8 focuses on the research hypotheses. 



3.1 A cognitive psychological framework 

An essential part of the theoretical framework is based on considerations about the 
cognitive functioning. We opted for instance for the componential theory of Sternberg (1985, 1988) 
to describe and explain cognitive functioning in our specific distance education context. Sternberg's 
cognitive processing approach sti-esses, in one consistent framework, the importance of a large 
complex of cognitive processes. 
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Cognitive functioning is - according to Sternberg - described with the concept 'components'. A 
component is "an elementary information process that operates upon internal representations of 
objects or syinbols" (1986, p225). Sternberg distinguishes three groups of components: 

- Performance components (PC); 

- Knowledge acquisiticr, components (KAC); 

- Metacognitive components (MC). 

Metacognitive components control, evaluate and plan the activities of the other components. They 
are executive processes. The knowledge acquisition components are involved when new 
information is integrated in already available information in memory. This integration process 
implies encoding, combining and comparing. Performance components t^/e those processes that 
operate when a task is actually executed. Knowledge acquisition and performance components are 
non-executive processes. The metacognitive components can be perceived as "the stage directors 
that tell the actors - the performance and knowledge acquisition components - how to act. The 
actors in turn provide feedback to the directors about how the show is going." (Sternberg, 1988, 
p.l89)". 

ESD have a support function for the learning process (PC, KAC, MC). They aim at scaffolding the 
learning process by supporting orientation, planning, processing and evaluative aspects of the 
individual leai*ning process. 



3.2 Reading comprehension: an important intervening process 

As stated earlier, the interference of the reading process in our research cannot be neglected and is 
even to be incorporated into our model. We focus on 'discourse comprehension' and not on the 
decoding aspects of the reading process. This decoding process is, for experienced adult readers 
an automatic process, that probably hardly causes individual differences, although there might be 
an indirect influence. Though at this level reading and listening have become very similar (Kintsch 
& van Dijk, 1978; Perfetti & Lesgold, 1977), we will only focus on the comprehension of written 
texts (discourse). 

In this tex part, we restrict ourselves to an introduction into recent and main developments in the 
area of discourse comprehension to be able to relate these reading theories to our theoretical 
model. 



3.2.1 Discourse comprehension 

A major publication in the discourse comprehension literature is Kintsch & van Dijk (1978). In this 
publication, they present the model described below. The model shows how a reader builds 
coherent global text meaning from isolated text propositions. The rules, followed through this 
process are called "macrorules". This model, with minor or major adaptations is still used and still 
influential in current research and literature (e.g., Rayner & Pollatsek, 1989; Britton & Gulgotz, 
1991; Oostendorp en Peeck, 1991). 

Discourse comprehension is a complex process that still is not comprehended by theorists (Rayner 
& Pollatsek, 1989). In recent attempts computer models are built to simulate the process. These 
simulation programs remain restricted and can be compared to the still imperfect attempts to 
automate the translation of texts. Good translation requires understanding of the text. 
Obstacles in theory construction and research of discourse comprehension is the measurement 
issue and speculations about the number of 'factors' that should be used to describe this reading 
skill. Recent theorists adhere the view that the skill is based on a single factor, called 'verbal 
intelligence', causing individual differences (Rost, 1989). Despite the heavy debates on this issue, 
there is a certain agreement about the - minimal set of - 'building bricks' constituting a model for 
discourse comprehension. Kintsch (1988, p.l63) : "Discourse comprehension, from the viewpoint of 
a computational theory, involves constructing a representation of a discourse upon which various 
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computations can be performed, the outcomes of which are commonly taken as evidence for 
comprehension. Thus, after comprehending a text, one might reasonalily expect to be able to 
answer questions about it, recall or summarize it, verify statements of it, paraphrase it, and so on.". 

At this level, we already perceive cognitive processes that can be related to components of 
cognitive functioning as described by the Stemberg-model. 

Kintsch continues : 'To achieve these goals, current theories use representations with several 
mutually constraining layers. Thus, there is typically a linguistic level of representation, 
conceptual levels to represent both the local and global meaning and structure of a text (e.g., the 
micro- and macrostructure, constituting the text base in van Dijk & IGntsch, 1983), and a level at 
which the text itself has lost its individuality and its information content has become integrated 
into a larger structure (e.g., van Dijk & Kintsch's situation model)." 

Figure 1 gives a schematic overview of the overall construction and functioning of the discourse 
comprehension model as found in literature (Ballstaedt, 1981). In this figure, the text is the input of 
the process and the cognitive schemata form the output. The micro- and macrostructure of the text 
represent subproducts of the comprehension process. 

The process starts from a 'texf . This text is decoded : words, meaningful units are extracted from 
the text. If a text is comprehended and letters/words are turned into meaningful, but still isolated 
units (rrucropropositions), these units are combined and a trace is constructed that runs through 
the text. 

The decoding process is done at local text level; small text parts are taken step by step. When 
words have become meaningful units fljut still isolated), these units are combined and the trace 
that runs through the text is reconstructed. This implies a certain reduction of the initial text 
content. Moreover, in order to comprehend the text, the reader does also have to know even more 
than the actual text content. Inferences and elaborations have to be made. Inferences are text 
additions constructed by the reader; elaborations are inferences that go beyond the rrunimal 
information needed to understand the text (e.g., the reader makes a pictorial representation of the 
main character of the story). Finally, these processes result in a set of cognitive schemata about 
the text content. 

It is taken for granted that the process, described above is of a cyclic nature. Text parts are 
processed part by part. At the end of the cyclic process, the complete text is comprehended. It is 
also important to note that there is no strict hierarchy in the processes and building bricks 
described. All processes influence each other. Of course understanding a text starts with reading 
the first sentences of a discourse ('bottom-up') but always immediately expectancies, schemata and 
ideas from the reader start to play an important role (top-down). Moreover, the interpunction of 
the overall process is not unidirectional; text comprehension does not only start from a newly read 
text (bottom-up), but is also initiated from existing and available cognitive schemata (top-down). If 
people read for instance some new text, that is consistent with their content expectations, the text 
will be read and comprehended at a much faster rate, compared to texts with unexpected content 
parts. Taking into account these latter remarks, it is to be stressed that the model in Figure 1 is a 
simplification of reality. The layered structure and process flow suggested in the model, represent 
rather 'one' possible conception of the comprehension process, compared to processes and the 
structure in real life. This strong interrelatedness of structure elements, layers and processes is 
depicted in Figure 1 with arrows : the large arrows represent information flows. The curved 
arrows suggest a possible process run. 

It must be stressed again that there is no agreement yet nor strong evidence for any of the 
developed models of reading comprehension. There is probably still a long way to go before the 
process is really understood. 

Figure 1 represents the complete scheme of the comprehension process. The outcomes of the 
process are represented as 'bricks', the process elements are depicted with ellipses. For a 
description of some otl\er models of reading we refer to Rayner & Pollatsek (1989). 
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Figure 1: Schematic representation of the building bricks and processes in the comprehension 

process (based on Ballstaddt, 1981) 



3.2.2 Reading and learning 

Parallelisms between reading and learning can be found at the level of the nature of input and 
output (1) and at the process level (2). Differences are striking when the 'intentionality' and 
'finalit/ of both processes are clarified. 

-1- Parallelism between the nature of input and output of the learning and comprehension process. 
When recapitulating the comprehension process described above, and looking at the learning 
process in the context of distance learning, based on written study materials we see that both 
processes ma inly start from a 'texf . At the output level, both processes result in the construction 
of 'knowled';;e' (comprehension). Following ihe Sternberg model the knowledge acquisition 
components result in coded, combined and compared representations. The comprehension process 
described earlier results in cognitive 'schemata'. These schemata are comparable to the 
'representations' in our conceptual frame of reference. But of course, since students, when 
learning, are expected to go beyond the mere text content by answering questions, executing tasks, 
etcetera. The final output of the learning process will be much more complex. 

-2- Parallelism in the process aspects of the learning and comprehension process. 
Since learning from text is based on comprehending the study text in order to extract meaning, in 
order to perform specific learning processes there is an overlap between both processes at 
consecutive stages. 

If we concentrate on the - in time and function - overlapping cognitive processes between learning 
and reading (comprehension), we can state that the processes described along the Kintsch and van 
Dijk model are comparable to the processes in the Sternberg model. Figure 2 shows the possible 
overlap between both processes in the course of a learning process. Since both processes, in 
starting from a text, are directed towards meaning construction, the three groups of cognitive 
components can be used to characterise the subprocesses described in the Kintsch & van Dijk 
model. In reading, metacomponents are active: there is the problem to be recognized and solved. 
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its extent is to be stateu, attentional resources are to be allocated, the process is to be monitored, 
specific strategies for task performance are to be selected, etcetera (cf. Markman, 1979). The 
knowledge acquisition components are in game when the cognitive schemata are build, step by 
step, by encoding, combining and comparing (cf. the macrorules of Kintsch & van Dijk). 
Performance of the selected strategies for task performance is to be guaranteed by encoding, 
combining and comparing (text) stimuli and applying task specific activities. 

- Differences between the comprehension and learning process. 

As stated earlier, there is not a complete overlap between the reading and learning process. 
Learning goes beyond mere reading and comprehending since the learning activated in the text 
goes beyond the mere text content. Students are expected to solve questions, execute tasks, look 
for resources or extra documentation, etcetera. 

Also the kind of reading, implied by learning is in a way different from reading in general. The 
type of reading on which learning is based differs from other types of reading by its special 
'intentionality'. Reading for studying is for instance different from reading as a leisure activity or 
reading to search for specific information (van Hout Wolters & Kerstjens, 1990). Reading, when 
linked to learning can be considered as a kind of extended reading. Just & Carpenter (1987, p. 
424) state the interrelation between reading and learning as follows : "Many study strategies can 
help readers understand the texf s content and organization and relate it to what they already 
know, including rereading, outlining, and networking. These strategies are dosely related to 
normal comprehension processes; they require elaborating on and making explicit the kinds of 
informational structures that readers construct during comprehension.". 
So, the difference between both processes is to be found in the intentionality of the reading 
process. Oostendorp & Peeck (1991, p. 153) describe this as follows^ : "...when reading there is not 
always a learning intention. This means that the process is not directed towards changing a 
cognitive structure. Learning is always directed towards changes in cognitive structure.". The 
special intention of learning is also found in the larger stress that is put on the incorporation of 
knowledge representations (cognitive schemata) into the memory structure in order to remember 
what has been read. Many researchers (Gemsbacher, 1985; Lorch & Lorch, 1985; Mandler & 
Johnson, 1977; Thomdyke & Yekovich, 1980; etcetera.) stressed this special interrelation between 
comprehension, learning and remembering. 

In summary, learning can in part be considered as a special case of discourse comprehension that 
is focused on changes in knowledge acquisition (representation and retention). 
Booiunan & Kok (1986) adopt a similar view and have constructed an extended reading model 
(Figure 2). The interesting thing about tlieir model is that they offer us the opportunity to 
anticipate on our discussion about the potential role of ESD in learning from text. Next to technical 
reading and text comprehension they discern 'reading for studying'. Here, the learner /reader goes 
beyond the text and starts processes that are labelled as 'in-depth analysis' which imply repeating, 
transfomning and elaborating^. 



Translation by the authors of this article. 

Note the parallelism between their conceptual framework and the schema theory of Anderson in which 
learning is approached as a process during which schemata are transformed, tuned and refmed (Anderson, 
R.C., Reynolds, R.E., Schallert, D.L. & Goetz, E.T., 1977; Anderson, R.C., Spiro, R J. & Anderson, M.C., 1978). 
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Figure 2 Model of reading for studying (Boonman & Kok, 1986, p.69) 



3.2.3 Individual differences 

Since discourse comprehension is related to learning, reading ability does correlate to a high 
degree with educational achievement. Reading ability correlates with school achievement results 
and general intelligence indicators (Rost, 1989). Individual differences in reading ability are 
therefore an important intervening variable in the process of learning from text. In literature, (e.g.. 
Singer & Donlan, 1989) 'types' of readers are distinguished. If we look at the building bricks of the 
model for the reading process, differences between readers can be related to specific elements or 
process parameters. Oostendorp & Peeck (1991) have derived a long list of differences. It is also 
important to note that differences between readers can be important and are difficult to overcome 
(Rost, 1989; Singer & Donlan, 1989) : 

There are differences in the semantical and syntactical base. Less able readers possess for 
instance a less elaborated vocabulary. They do not only understand concepts to a lesser 
degree but the slower and non-automatic word recognition process invokes an extra 
bottleneck by putting too heavy demands on the capacities of the short term memory (cf. 
also Perfetti & Lesgold, 1977). 

Differences in grammatical knowledge hinders of course text comprehension. 
Differences ir speed and accuracy by which concepts are recognized and put in context. 
Differences in prior knowledge (cf. Dochy, 1992) 

Differences in cognitj' e functioning (components in the Sternberg model). 



Next to differences in relation to cognitive aptitudes. Oostendorp en Peeck (1991) also mention 
differences that go beyond the mere cognitive aptitude differences : 
Differences in interest. 

Differences in motivation (differences in attitudes towards the text content). 

As stated before it should be noted that these differences do not occur isolated. Individuals lacking 
one of these 'abilities' usually lack others. 
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Z>3 The task environment 

The task environment refers to those variables that define the setting in which the student studies. 
These variables show a great variability in a distance education setting. 

Learning, in this theoretical framework, is the result of the interaction of an individual with a task 
environment as defined by the distance education setting in which the student is expected to attain 
objectives in a self-instructional way, by adopting an independent learning approach. 
The task environment can have an influence on the cognitive functioning and the reading process 
for instance by means of the learning materials provided to a learner. In distance education in 
most cases the learning materials consist of printed self study materials. Other task environment 
variables can vary of course a lot across learners since there is no 'control' over these variables n 
distance education. It is possible that some students have more problems with distracting 
environmental influences then others. 



3.4 Individual variables 

The set of potentially relevant individual variables is considerable. For this research we limit this 
set of variables to the next three subsets. 

cognitive functioning: 

There are many intra-individual variables related to differences in cognitive functioning. 
Some examples are prior knowledge (Dochy, 1992) and crystallized or fluid abilities 
(Lohman, 1993). Prior knowledge can be an important predictor of study outcome. 
Differences concerning cognitive functioning are important intervening variables between 
the input and output of the study process in distance education. Note that these abilities 
are not necessarily stable and unchangeable. It is evident that prior knowledge can change 
but this also seems to be the case with intelligence tests. Lohman (1993, p.l3): 'Intelligence 
tests, like achievement tests, measure developed abilities, not innate capacity or potential.' 

reading comprehension: 

A particular example of an intra-individual difference in cognitive hinctioning that can 
influence study outcome and one that is strongly related to general intelligence ('G') is 
reading comprehension. 

studying texts with/without ESD: 

It has been found that studying texts with ESD has important advantages to students as 
stated earlier. Comprehension improves and examination results are better. There might 
however be a relation with individual variables, in other words: do some students benefit 
more fiom ESD than others, or do some students use ESD more than others? The research 
conducted so far has not revealed important differences in the effects of ESD although some 
groups of students (for instance female students) tend to use ESD more than others. 



35 Non-cognitive aptitudes: motivation 

Studying with self study materials presupposes a strong intrinsic motivation of students to get and 
stay involved in the study process. Most Chi-students combine their study with a regular job and 
other duties/tasks for instance as head of a family. Motivation is therefore an, although often 
neglected in Ou-practice, important variable. 

Differences in motivation do not only occur during reading but can influence the complete study 
process (Boekaerts & Simons, 1993). Motivation is a very complex concept. In fact motivation 
stands for the mental processes that determine what behaviour an individual performs, how long 
and with how many effort. So 'motivation' is as a concept almost as broad as psychology itself. 
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There are many aspects related to it in education like for instance the concept of internal vs. 
external motivation, reinforcement and fear of failure. In this research motivation does not play a 
major role so we will not give more attention to the theoretical underpinning of this concept in our 
research. 



3.6 Integrating processes and variables 
Figure 3 summarizes the theoretical framework. 




Figure 3: A model for learning from printed study materials 



The model in Figure 3 depicts the interaction of the processes described above. In the next section 
we clarify how the specific research method adopted in this research 'Eye movement registration' 
builds on this theoretical framework. In part 3.8, we derive from the framework the basic 
hypotheses for our investigation. 



3.7 Eye movement registration research 

As explained earlier, eye movement registration was introduced in this project, as an 'extra' to 
traditional methods, such as questionnaires and interviews. These methods are considered to 
depend heavily on reflection/introspection and are thus a possible source of subjectivity. Eye 
movement registration is used as a supplement, not as a substitute for traditional methods, because 
it has its own problems. One major problem of eye movement registration is that the relation 
between eye movements and higher psychological functions, such as discourse comprehension, is 
far less clear than its relation to lower order functions, such as lexical access and word recognition 
(Rayner and Pollatsek, 1989). However, objectivity is an important virhie of eye movement 
registration, which makes it worti\while to explore it as a tool to investigate the use and effects of 
ESD. 

Eye movement registration is used in various research areas ranging from reading research and 
picture scanning to oculomotor conti-ol. In the context of educational research it is still mainly used 
to shidy tiie reading process (van Lieshout, 1982). Initially, it focused on local aspects of ^he text 
processing, such as the impact of visual/graplucal features or word frequency. McConckie and 
Rayner (1975) for example tii.-d to determine how peripheric (parafoveal) text influences ti\e 
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processing of the text in and near the fixation point of the eye. 

Increasingly however, researchers became interested in eye movement patterns to study 

comprehension processes. 

Mandel (1979) and Shebilske & Fisher (1981, 1983) revealed for instance differences in eye 
movement patterns when reading specific text parts which express key ideas. 
Rothkopf & Billington (1979) discovered differences in eye movement patterns when 
reading text parts which differ in importance according to the objectives of the reader. In 
this research set-up students were asked to memorise specific objectives, before reading a 
text. Reading of goal-relevant sentences resulted in over twice as many and longer 
fixations (> 15 msec) than less relevant sentences. 

Just & Carpenter (1980) used data from eye movement registration to construct a model to 
describe reading comprehension. 

Grabe, Antes, Thorson & Kahn (1987) investigated the pattern of eye movements produced 
when adult readers encounter cross-sentence contradictions. Uninformed subjects were 
asked to prepare the answer to a question that would follow each paragraph. Informed 
subjects were told that the paragraphs could contain contradictions and that they had to 
report the contradictions and to answer a question after each paragraph. Analyses of eye 
movement patterns provided evidence that both informed and uninformed subjects were 
aware of the inserted contradiction but that instructions to search for inserted errors 
caused readers to alter their reading bdiaviour. 

Blanchard & Iran-Nejad (4987)-found that there was an increase in the number of fixations 
associated with rerieading surprising lines of stories when compared to the same lines in a 
control condition, where these lines were not surprising. 

Vauras, Hyorw and Niemi (1992) examined the effect of text structure on the reading 
process and recall. Each subject read and studied a coherent and incoherent text 
Incoherence was brought about by changing sentence order in central paragraphs of a text. 
Immediate free recall followed this reading/study process. The eye movement data 
showed that structural incoherent text segments attracted the largest number of regressive 
fixations, and hence were given more visual attention than coherent text segments. On the 
other hanrf, more rereadings were devoted to text segments resolving the incoherence. 
Rayner and Pollatsek (1989) presented figures for fixation duration, saccade length, percentage of 
regression and the number of words per minute for 10 good college-age readers reading different 
types of text. We have extended these figures with the reading time per word in order to facilitate 
comparison with our own results. Table 1 shows the extended figures of Rayner and Pollatsek. 
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Table 1: Reading characteristics for 10 good college-age readers reading different types of text 
(adapted from Rayner and Pollatsek, 1989) 



topic 


fixation 
duration 


saccade 
length 


regressions 

(%) 


words per 
minute 


per word 


light fiction 


202 


9.2 


3 


365 


164 


newspaper 
article 


209 


8.3 


6 


321 




history 


222 


8.3 


4 


313 


191 


psydiology 


216 


8.1 


11 


308 


194 


english 
literature 


220 


7.9 


10 


305 


197 


economics 


233 


7.0 


11 


268 


223 


matliemalics 


254 


7.3 


18 


243 


247 


physics 


261 


6.9 


17 


238 


252 


biology 


264 


6.8 


18 


233 


257 


mean 


231 


7.8 


11 


288 


208 



In general, research within cognitive psychology on language processes has been focused on the 
processing of linguistic information alone (Duffy, 1992). However, most things we read consist of 
text in combination with other graphical elements, such as pictures, photographs, schemes and 
tables. Language comprehension of text-picture combinations can be regarded as a new area of 
research. Recently, a number of studies have become available that start to explore this new area in 
relation with eye movements (Hegarty, 1992; Can oil, Young and Guertin, 1992). These 
developments seem relevant in the context of ESD, in general learning material also consists of a 
mixture of text and other graphical elements. 

The studies mentioned above, indicate that the spatial aspects of eye movement, such as the spatial 
distribution of fixations of a page of study lateral, are at least equally important as temporal 
aspects, such as fixation durations and gaze durations. 

The eye movement method surely also has disadvantages. The special research set-up affects the 
ecological validity of the setting. Subjects are asked to study and read in a special setting, 
including a special chair, head-support and are often not allowed to move their head when 
reading. Although, it v^all be shown that subjects did not experience the set-up as disturbing. In 
our approach, the technique has been chosen to confirm/disconfirm findings of earlier research, 
which was based on interview and questionnaire techniques. The objectivity of the data obtained 
with the eye movement registration method would be a valuable counter-balance for the 
'subjectivity' of the data source in the earlier research methods adopted. 



3.8 Deriving research hypotheses 

Previous research on ESD resulted in conclusions we now seek to confirm/disconfirm with the eye 

movement registration method: 

ESD are used at a deep level. Students do not only read them but they use them in view of 
their study process and directed towards further text comprehension (Poelmans et al, 1992 
and Valcke et al, 1993); 

When ESD are used at deep level and not just read superficially, this should be measurable in 
differences in eye movement patterns. The present research should therefore reveal that deep level 
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use of E5D can be measured by looking at spedfic reading times and reading times per word. In 
this research we cannot make use of number of fixations and average fixation length on text parts 
incorporating ESD because we use copies of printed text materials that contain too many text per 
window to make accurate estimates of these rates. 

Differences in reading times and reading times per word can be expressed by negative and/or 
positive values. Certain ESD can invoke for instance longer or shorter reading times; this depends 
on the spedfic function/effect of the particular ESD researched. An BSD can imply that one only 
concentrates on a certain part of the text ir\formation and 'skips' the rest of the text by reading at a 
faster rate; e.g., when dealing with 'examples'. 

We expect the following ESD to evoke longer total readings times and reading times per word: 
learning objectives (because subjects know they will be tested with a test based on the learning 
objectives); titles; margin texts (this is text that is stressed by margin texts, the reader knows that 
this text is of prime importance); text in italics; questions. 

BSD that are expected to be read shorter: content page; introduction; examples; summary. 
The ESD 'feedback' is expected to be read at an average rate. This depends on the information in 
the feedback. If it is new or unexpected for students it will be read longer per word, else it will de 
read shorter per word. 

Schemes cannot be compared with 'standard' text because there is no adequate procedvu^ to define 
the length of a standard text schemes are to be compared with. But, despite this inconvenience, 
this type of ESD is expected to produce 'jumps' in the reading process of a text in which they are 
embedded. 

These basic arguments can be reformulated as the following three sdentific hypoflieses: 

1. Comparing the reading time when reading ESD or standard text exhibits no 
significant differences. 

2. Reading ESD or standard text produces no differences in reading times per word. 

3. While reading 'schemes' there are no 'jumps' into the text. 

Originally, building on the theoretical framework, also the influence of extra intra-individual 
variables on the use of ESD was predicted: Do poor readers use ESD in a different way compared to 
good readers? Are poor readers less well aware of functions of ESD and are they less well able to 
distinguish between basic information and 'examples' in study texts? 
Unfortunately, the amount of students who volunteered for the eye movement registration 
research was too small. Consequently we were not able to study this research track. We did use 
however these variables to check for characteristics of the research sample. 
This research will have an explorative function in the first place. We will pay much attention to 
the general experiences on working with this system and we will, in some rases, prefer qualitative 
analysis of data to quantitative research of data with statistical significance levels. 
'Nevertheless a set of scientific hypotheses is necessary to investigate the 'powef of the technique 
to reveal relevant differences in eye movement patterns in relation to ESD. 

4 Research design 



4.1 Research sample 

First year law students were invited to participate in the study. A letter was sent to those students 
who wished to partidpate, confirming the date and place for the research. Eleven students 
participated. The sample consisted of 5 male and 6 female students. All students had prior 
knowledge of Governmental law. In section 4.7 statistical evidence is put forward that indicates the 
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representative nature of the research sample. A special condition to participate was p'-.t forward 
since subjects had to have normal uncorrected visual abilities. Wearing glasses or lenses would 
have resulted in unreliable eye movement monitoring. 

The reader may expect that students of the Ou of the Netherlands participate in this study. But, 
students in a distanc<? education setting have difficulties to get involved in a research settii;g; 
especially because of their combination of work and study and the fact they live at long travelling 
distances from the institute. For research purposes, we therefore selected a comparable sample of 
law students taking courses at a non-distance education university, the university of Limburg 
(RUL). Despite this differing setting, these students have experience with self-study and self-study 
materials because the educational university system builds on 'problem oriented learning'. 
Moreover, before the experiment the students were given an introduction on the organization of 
Ou of the Netherlands course materials and the functions and effects of ESD. This was attempted to 
augment the correspondence of the actual research sample with the Ou-student population. 
Students involved in the research, received a financial reward for participating. 



4.2 Research instruments 



4.2.1 Eye movement apparatus 

The eye movement registration set-up contaii\s a number of different components: a slide projector 
to project study texts on a screen in front of the subjects; the EYECATCHER, this is the actual eye- 
tracker, which samples the eye position every 20 milliseconds; the samples are logged using a 
data-acquisition computer; the eye movements are superimposed in iieal-time on an image of the 
study text and recorded on a VCR simultaneously; a video camcorder is used to record the overall 
set-up. 

The projection screen is positioned at a distance of ± 80 centimetres. The slides represent almost 
exact copies of pages of a learning unit of the Ou-course 'Introduction to governmental law'. Each 
slide represents two pages. Since the learning unit consists of 15 pages, 8 slides are presented. 
Subjects can sv«tch between slides like turning pages in a book v«th a remote control. 
The EYECATCHER uses the corneal reflection/pupil centre method, which means that it calculates 
the fixation position from an infrared video image of the subject's eye. While reading, one eye of 
the subject is illuminated using an infrared radiation enutting diode (infrared LED). Infrared 
radiation is invisible for the human eye. The eye is monitored by an infrared-sei\sitive video CCD 
camera (For more technical information we refer to Stolk et al, 1991.). 

A special computer program was used to convert the eye position samples into fixations. Fixations 
are characteiized by their position and duration. Another program was used to analyze the eye 
movements. One option of the program is to reconstruct the actual scan-path of the subject. Figure 
4 gives an example of such a 'chronological path'. The figure shows the path the eyes follow while 
reading/studying the text. The black squares in the figure represent sets of two or three 
subsequent fixation points. The bigger the squares, the longer the sum of the fixation durations has 
been. 

However, there was one important problem in our way of using the EYECATCHJiR. The 
resolution of the EYECATCHER and similar systems is about 1 degree of arc, wh?ch equals 1,4 cm 
at a distance of 80 cm. The average distance between two subsequent words in the projection of 
the study text was below this value. If we assume that fixations fall in with subsequent words, it 
means that in this case it is impossible to reconstruct fixations from the original data set. It is well 
possible to reconstruct how long the eyes stay in larger text areas, such as sentences, sections and 
subsections. 
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Figure 4: Pattern of eye movements during studying two pages of a law course 




Figure 5: copy of a slide used in the research 



4.2.2 Reading comprehension test 

In the discussion of the theoretical base for this research, we already indicated that reading 
comprehension is a central component of the model for learning from printed study materials. 
Reading comprehension can be measured. To realize this, different alternatives are available: 
-self-ratings. 

The student is asked to value his own reading comprehension; for instance: "How many 
pages do you read in one hour?"; "Do you skip an article in the paper, because you didn't 
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understand it?" 

Answering these questions is not easy. They can be interpreted in different ways. It is also 
possible that students answer in a socially desirable way. 
administration of an existing reading comprehension test. 

No Dutch reading comprehension test is yet available at acadeniic level. Research is 
currently done in this field by de Geus, Henneken & Reitsma (1992) but no 
results /instruments are available at this moment. 
a test in a different language. 

This implies translation of a test developed in a different language. This is not a desirable 
option since translation is difficult and affects the reliability of the new 'translated' test. 
developing a new test. 

Developing a new test seems to be the only acceptable alternative. For this and related 
research purposes, a reading comprehension test at academical level was developed. The 
test presented to the students 23 text fragments (e.g., texts of courses of the Ou of the 
Netherlands). Each text fragment was followed by two propositions. The student had to 
judge their congruence/discongruence with the text fragment. A try-out of the test resulted 
in major revisions. The new test version was again probed and adjusted anew. This 
procedure resulted in a final version based on 22 text fragments, each fragment contains 
two questions. This final version of the test had a reliability rate of .68. For a detailed 
description of the development of this test, we refer to Martens (in prep.) 



4.2.3 Prior knowledge state test Law 

To measure the prior knowledge state in relation to the law domain, a final test for the 'Basic 
course Law' has been adjusted. This was realized in close collaboration with a law-expert. The 
number of test items was reduced and several items were rewritten. The final test consists of 32 
items: 20 multiple choice questions and 12 true/?/false questions. The quality of the items is 
checked with a special checklist (Poelmans, Manens, Valcke, Dochy & Bastiaens, 1993). 



4.2.4 Questionnaire 

The questionnaire constitutes of three different parts: 

Part A contains general questions about study motivation, age, study habits, educational 
level, gender, etcetera. Motivation is measured in two ways: a self rating of one's motivation level 
and intrinsic versus extrinsic motivation.This information is used to compare the experimental vyith 
the control sample. 

Part B contains questions about the course text the student had studied. They were asked 
to judge the clarity and difficulty of the text on a five points scale. These ratings helped to 
calculate the Learning Access Level (LAL) of the text version. 

Part C includes questions about the specific embedded support devices. Each embedded 
support device is reviewed. The questions focus on the use of the support devices, their functions 
and effects and their appreciation by the student. 

42.5 Comjjrehension test Governmental Law 

The comprehension test measures the mastery of the objectives pursued by studying the specific 
learning unit. The original version of the learning unit contained a comprehension test based on 
open-ended questions. These questions were used to develop a new test for this research. The 
quality of the items was inspected with a special checklist (Poelmans et al, 1993). The final test 
consists of 23 questions: 16 true/?/false questions and 7 multiple choice questions. When 
analyzing the questions from a didactical perspective two subsets can be distinguished: 
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'knowledge questions' (13) checking the straightforward recapitulation of the content; 
'insight questions' (10) measuring a more thorough insight in the domain. 
Tlie test was representative for the learning objectives pursued with the learning unit. 



4.3 Research procedure 

The research took place during two half day-periods. 

The first half-day period 47 students participated. The location was an auditorium of the 
university of Maastricht. The reading comprehension test, the prior knowledge state test 
and part A of the questionnaire were administered. After the session the students could 
choose between two research conditions: 

* studying a printed version of a learning unit (results of this research are reported 
in Martens et al, 1993). 

* studjnng in the experimental set-up with eye movement registration. 
Eleven persons volunteered to participate in the eye movement registration experiment. 
In section 4.6, tiie statistical analysis shows that - despite its small size - this experimental 
sul Troup of students is representative as compared to the larger group of 47 students. 18 
fron. the group of 47 students studied the samt study material as our experimental group 
but with a printed version of the text instead of a projected version as used in the eye 
movement registration exjjeriment To summarize: 47 students filled in tests, form these 47 
18 students studied the material fom\ paper and 11 from slide projections. 

Eye movements registration procedure took place at the Open university in Heerlen. Before 
studying a learning unit of the course Introduction to Governmental Law' the subjects 
could read a short introduction to the learning unit. This introduction positioned the 
learning unit in its larger course-context and explained basic functions of ESD. This was 
necessary to deflate some differences with Ou-students, who by experience know about 
ESD. 

Students were informed about the procedure. They also were told that the procedure 
would end with a comprehension test about the learning unit studied /read. 
The procedure started with a calibration procedure to adjust the eye movement registration 
equipment to the individual student. This procedure requires the subject to fixate twenty 
calibration points, that are presented in four rows of five crosses on a calibration slide. 
The pages of the learning unit were also shovm using a slide-projector. The student saw on 
each slide two pnges such as he would see when reading from the printed version. The 
subject operated the slide-projector by means of a remote control, learning was self-paced. 
Returning to previous pages /slides was allowed. The students were asked to study as in a 
normal situation. During studying an infra-red eye-tracker registered the eye movements 
of the student. Researchers made annotations about the sequence/order of the studied 
pages. Three researchers were involved, each with a different task during the experiment. 
After studying the learning unit, students were asked to answer the comprehension test 
and part B and C of the questionnaire. To motivate the students an extra financial reward 
of /25 (Dutch Guilders) was prorr^i^od to two students with the highest comprehension test 
scores. A short interview concluded the experimental procedure. 



4.4 Analyzing the registered data 

The study text contains ESD. Fifteen areas in the text were defined, each subject to a specific 
different ESD. Some of these areas are shown in Figure 4. For each of these areas a similar text area 
was selected, that was not subject to an ESD. This resulted in area pairs, in which both elements 
had the same number of words. One element was subject to an BSD, the other was not. Both areas 
of a pair had the same position on a page. This enabled a comparison between text with and 
without ESD. 
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From the stored data, the ANALEYES software (Stolk, Smulders & Pelsmaeker, 1991) produced: 
the total reading time for each text area (in milliseconds); 

the number of jumps between text areas and schemes. When students read schemes they 
sometimes jump to the text and vice versa. This does not include the initial reading of the 
scheme. Only 'real jumps' to and from the text were counted. 

The data set was entered into the computer to be processed with the sps^ and SPSS for VJindows 

statistical packages. Scores for specific variables were processed as follows: 

Raw scores for the prior knowledge test, the reading comprehension test and the 
comprehension test Governmental Law were computed. This resulted in scores for three 
variables: total score prior knowledge, total score reading comprehension and total score 
comprehension Governmental Law. The comprehension test governmental Law consisted 
of 'knowledge' and 'insighf items. Sub-total score for 'knowledge' questions and 'insighf 
questions were computed. The variables that gave an indication of the use of a specific 
embedded support device were as well computed. The new variable gives an indication of 
the general use of embedded support devices. 

The variable 'student rating of Leanung Access Level' (LAL) was constructed by combining 
the ratings of the different items in questionnaire B. 

45 Reliability and validity of the research instruments 

In analyzing the validity and reliability of the research instruments, the following considerations 
were regarded: 

All items of the reading comprehension test should measure the same aptitude (internal 
consistency). Items that correlate negatively shotild not be part of the test. Cronbach's 
alpha of the remaining items should be at least .65. 

The compreheitsion test and the prior knowledge test consist of two subsets of items 
testing different mastery levels (knowledge versus insight). Internal coitsistency is therefore 
not expected to be high. Moreover, the test items cover very different law topics; thus 
affecting again internal consistency. 

We discuss the results by reviewing each research instrument separately: 
Reading comprehension 

Deleting test items who correlate negatively with the total score, 29 items remain in the 
final test Crombach's alpha for this final test version is .68 (standardized). 
Learning access level (LAL) 

A total score of the student rating of learning access level (LAL) is computed by adding all 
scores of part B of the questionnaire. No item-analysis has been performed. 
Prior knowledge test 

We already indicated that internal consistency is expected to be low, considering the 
nature of the test. 

The content validity was ensured by choosing /adapting an existing test in cooperation 
with an expert in the law domain. 
Comprehension test 

We repeat the problem mentioned earlier in relation to the prior knowledge state test 
Internal consistency is expected to be low, considering the nature of the test. No further 
reliability analysis has been performed. 

The content validity was also guaranteed by choosing/adapting an existing test in close 
collaboration with a law expert. 
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4.6 Checking for distracting aspects of the research set up 

We already mentioned that the research set-up can be questioned when considering ecological 
validity: 

The subjects had to study from slides in a dark room with dimmed light in the presence of 
at least two researchers. The devices used for projection, eye movement registration and 
processing were substantially and quite 'dominantly' present. 

Reading from slides differs from reading from paper. One subject mentioned problems 
with text readability. After decreasing the distance to the screen he was able to read the 
text. 

Another problem is that the subjects knew that the researchers were able to see what they 
were reading. This may have influenced their normal reading style. 
It could take quite some time (up to 35 minutes) before the eye-tr"cker all was calibrated 
rightly. During this period the subject had to wait. 

The chair of the experimental set-up was a so-called Rontgen-chair wath which the subjects 
were unfamiliar. 

In practice the research set-up proved to work better than we had foreseen. After a short period 
the students were used to the special setting and seemed to work in a concentrated way. By means 
of a short interview the students were asked afterwards to react to the set-up. All subjects 
indicated that they studied in a quite normal way and that they experienced no particular 
problems with this set-up. The only exception was the student who mentioned readability 
problems which could be solved before the experiment started. 

In order to check for the distracting influence on the learning process we also compared learning 
outcomes from the research sample with the outcomes of students studying the same text, but as 
printed on paper. These students were from the same university, the same study year and they 
studied the same subject as our research sample. 

Table 2: Learning outcomes from research sample 'slide' versus sample 'paper': t-tests, two-way 
probability 



variable 


group 'eye'(n=ll) 
average score 


group 'paper' (n=18) 
average score 


D.F. 


t-value 


comprehension test 
total score 


14.73 


15.14 


45 


-.36 


knowledge questions 


8.7 


8.14 


45 


.39 


insight questions 


6.0 


7.0 


45 


-1.41 


use of ESD 


7.55 


8.33 


27 


-1.77 


rating LAL 


44.0 


46.33 


45 


-1.06 



* p< 0.05; *' p< 0.01 



Table 2 indicates that students in the research set-up do not differ from students in a normal study 
condition. Slide projection does not significantty influence comprahension scores, self ratings of use 
of ESD, nor ratings of learning access level when compared to learning from a 'normal' printed 
version of an Ou course. 

However, some caution is still appropriate. Although none of the t-tests produced significant 
results, a trend can be perceived. Four out of five comparisons between the two research samples 
are in favour of learning from a printed version (negative t-values). therefore we conclude tiiai 
there might be a small, negative influence from the research set up with slide projections, although 
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this influence is too small to raise significant differences. 
4.7 Checking the characteristics of the research sample 

The n volunteers in this research can be compared to a larger sample of 36 students who 
participated in a parallel research set-up where their approach to studying form text was analyzed. 
The two samples were compared on seven variables: educational level, self rating of motivation 
(two measures), prior Icnowledge, reading comprehension, sex and age. The results of the t-tests 
are presented in Table 3. 

Table 3: Comparisons between characteristics of research sample with other sample: t-tests, two- 
way probability 



variable 


group 
'eye'(n=ll) 
average score 


group 'paper'(n=36) 
average score 


D.F. 


t-value 


educational 
level 


2.2 (n=10) 


2.44 


33.05 


-1.21 


self rating of 
motivation (1) 


1.18 


1.08 


45 


.92 


self rating of 
motivation (2) 


3.55 


3.95 


12.47 


-1.34 


prior 

knowledge 


10.27 


9.14 


45 


-.17 


reading 

comprehension 


11.56 


11.95 


45 


.87 


sex*~ 


1.55 ' 


1.55 


45 


-.06 


age 


20.64 


20.28 


45 


.55 



* p< 0.05; " p< 0.01; *" l=male; 2=female 



Table 3 shows that the 11 students do not differ significantly ft-om the other students m the larger 
research sample. The students who enroled voluntarily for the experiment of eye movement 
registration do not appear to be a 'special' subgroup of a larger research sample. 



5 Results 



5.1 General conclusions 

Some prelinunary remarks 

Due to a technical problem the data of one subject could not be used for further analysis. 
The reader may notice that 'comparable standard texf areas are not compared with each 
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Other in Table 4. This is not done because it is not directly relevant for the hypothesis 
testing. Another reason for not comparing standard texts is that they are not of the same 
length. However, to control for an -unexpected- influence of differences in reading time for 
standard texts, we made some comparisons between standard texts of about the same 
length, therefore, such areas - of about the same length - were compared with one another. 
No significant differences were found in total reading time. 

Considering these preliminary analysis results, we concluded that comparison at the level 
of ESD areas was acceptable. After a first analysis of the results we will further evaluate the 
usefulness of the comparisons between these text areas. 

Table 4 shows the average reading times in milliseconds of ten subjects on the fifteen predefined 
area pairs. The left column describes the areas that are subject to an ESD, the right column contains 
the results on areas similar to the areas of the left column, but that were not subject to an ESD. 
'Schemes' are dealt with separately (cf. Table 6). 



Table 4: Average reading times in milliseconds per window and pairwise T-tests 
(two way probability, N=10; 9 d.O 



variable 


registrations 




ESD 


standard text 


t-value 
total 




total 

reading 

time 


SD 


total 

reading 

time 


SD 


reading 
time 


content page 


6238 


6195 


14244 


7522 


-2.89' 


introduction 


9086 


3749 


14262 


7532 


-2.17 


learning 
objectives 


52432 


29907 


57096 


20783 


-.83 


title 


2118 


1814 


9201 


21114 


-1.04 


margin text 1 


26900 


8152 


21366 


7226 


1.84' 


margin text 2 


12426 


6929 


9796 


7615 


.75 


example 1 


42672 


14498 


45692 


14242 


-.57 


example 2 


23578 


6673 


31002 


13894 


-1.77 


example 3 


41082 


11125 


46938 


12382 


-1.43 


text in italics 


7274 


3693 


10418 


4644 


-2.11' 


question 1 


18788 


11762 


9729 


8563 


2.08' 


question 2 


22160 


11567 


6951 


3712 


4.11" 


summary 


25678 


16007 


35464 


13161 


-1.29 


feedback 1 


32998 


25492 


36500 


13761 


-.39 


feedback 2 


42518 


17262 


36500 


13762 


1.00 



■ p< 0.05; " p< 0.01 



When we divide the total reading time per window as printed in Table four by the number of 
words per window it turns out that the preliminary conclusions of equality of standard texts is not 
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correct. This means that the notion of comparable standard text based upon areas of expected 
normal reading is not valid in the way we have defined it. When calculating the total reading per 
word (see Table 5) it shows that the variably in reading time per word for these areas is too high. 
This is especially the case for the relatively small areas with only a feW words. This has two 
important consequences: 

1. Variables that are based on windows containing 20 or less words are deleted from further 
analysis ('title' and 'text in italics'). These two variables also have extreme high reading times per 
word (836 and 744 milliseconds) and very high standard deviations. These values are higher than 
any others in Table 5. We consider these measures as unreliable. 

2. The results as reported in Table 4 should be treated with caution. 'Standard' texts are not really 
standard. Standard text evokes differences in reading time per word and therefore windows 
containing standard text can not be considered equal. Therefore it seems better to construct one 
general variable to compare ESD with. This can by done be computing the mean reading time per 
word for all the data we have obtained (both reading times per word for ESD and for 'standard' 
text). This value is 344 milliseconds. 

Table 5: Reading times per word in milliseconds and comparisons with mean (binomial test; one- 
tailed probability, for feedback 1 and 2: two-tailed probability) 



variable 


registrations 




ESD 




comparable standard text 




numljer 
of 

words 


total 

reading 

time 


reading 
time per 
word 


binomial 
test, sign 


number 
of 

words 


total 

reading 

time 


reading 
time per 
word 


content page 


39 


6238 


124 


.** 


40 


14244 


356 


introduction 


50 


9086 


159 




52 


14262 


274 


learning 
objectives 


101 


52432 


519 


+ 


109 


57096 


523 


margin text 1 


56 


26900 


480 


+ 


70 


21366 


305 


margin text 2 


40 


12426 


311 




44 


9796 


22 


example 1 


166 


42672 


257 




168 


45692 


272 


example 2 


75 


23578 


314 




85 


31002 


364 


example 3 


134 


41082 


307 




132 


46938 


356 


question 1 


29 


18788 


648 


+ 


33 


9729 


295 


question 2 


40 


22160 


554 


+ 


43 


6951 


162 


summary 


92 


25678 


279 




92 


35464 


385 


feedbacic 1 


102 


32998 


323 




lOO 


36500 


365 


feedback 2 


104 


42518 


409 


+ 


100 


36500 


365 



' p< 0.05; " p< 0.01 



What conclusion can be drawn from Table 5? 

In Table 5 we look at the reading times per word. A comparison of the reading times per words in 
our experimental results with Rayner and Pollatsek's figures in Table 1 shows that in general the 
reading times per word are higher in our rase. 

ESD are read 360 milliseconds and standard texts 327 milliseconds. In Table 5 reading times per 
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word for ESD are compared with the average value of 344 milliseconds in the column 'sign'. A 
positive sign indicates that the reading time per word is higher and a negative sign indicates the 
opposite. Note that 'reading time per word' is not based on dividing the total reading time by the 
number of words for all cases together but that this is done for each case separately. From these , 
values the mean is calculated which is printed in the column 'reading time per word'. Also note 
that we make use of round values. 

We have performed statistical analysis on these signs with a nonparametric sign test (binomial 
test). We had to choose for a nonparametric test for this particular analysis because of non-normal 
characteristics of the variables we used here. However we opt for a more qualitative interpretation 
of the obtained data than an interpretation that is based on significance levels. First of all this is 
related with the explorative character of this study. Second with the limited number of subjects 
and on the third place does the use of a nonparametric sign test imply that a lot of information is 
lost. 

Although there is a lot of variance between 'standard texts' that we cannot explain and that is 
possibly due to content differences between these fragments of standard text, it is remarkably that 
the signs as printed in Table 5 are quite well in line with our hypotheses. We repeat the expected 
differences between ESD: 

longer total readings times and reading times per word were expected for: 
learning objectives; titles (not analyzed); margin texts (this is text that is stressed by margin texts, 
the reader knows that this text is of prime importance); text in italics (not analyzed); questions. 
ESD that are read shorter: content page; introduction; examples; summary. The ESD feedback is 
expected to be read at an average rate. 

ESD that were expected to evoke shorter reading times per word do so in most cases and ESD that 
were expected to be read longer per word also do so. There is one exception: margin text 1 has a 
positive sign and margin text 2 has a negative sign. The explanation for this may be situated in the 
content of the text fragment. We will not discuss this governmental law issue here. Feedback 1 has 
a positive sign and feedback 2 has a negative sign. This is quite well in line with the expectations 
since we expected feedback to yield average reading times. Both values for feedback do not differ 
much from the overall average of 344 milliseconds per word. 

5.2 Looking at schemes 

As stated earlier schemes have to be handled in a different way. Schemes cannot be compared 
with standard text because there is no adequate procedure to decide about the length of the 
standard text. Schemes are expected to evoke 'jumps' in the text. 

In Table 6 we counted the number of jumps to the text On average, each scheme evokes 3.6 jumps 
to the rest of the text. 

Table 6: Eye movement registration data in relation to schemes 



schemes 


number 
of 

fixations 


total 
reading 
time (msec) 


average 
fixation 
length 


jumps 
in text 


scheme 1 


86.6 


43402 


501 


3.7 


scheme 2 


24.7 


12735 


516 


2.5 


scheme 3 


45.7 


25770 


564 


4.5 
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5.3 Supplementary results 

The data we could collect with the program ANALEYES were extensive. In this section we will 
present some additive infonnation, that is not directly linked to hypothesis testing but that is, in 
our opinion relevant. 

To examine if all ESD have been read /used by the students, we controlled the individual 
data files. It turned out that every ESD was looked at, at least once by each student. In 
more technical terms: for each student there w£-j at least one observation per ESD. 
Inspection of video recordings of the eye movements and inspection of patterns of eye 
movements during studying the course (such as in Figure 4) showed important differences 
in reading style. We have not statistically examined these differences between reading 
patterns because the research sample was too small for such complicated analyses. 
The inspection of the individual data files also revealed that there are differences in study 
approach. Some students (n=5) tend to reread while others (n=5) only read the text once. 
By 'rehearsing' we mean that a subject reread at least one ESD. 

Because of the small sample size no comparisons were made between subjects, considering 
specific intra-individual variables (e.g., prior knowledge, gender, etcetera). 



6 Discussion 



6.1 Conclusions 

We repeat the 3 scientific hypotheses: comparing the reading time when reading ESD or standard 
text exhibits no significant differences; reading BSD or standard text produces no differences in 
reading times per word; while reading 'schemes' there are no 'jumps' in to the text. 

Considering the results, the following can be concluded: 

1 Hypothesis 1 can not be answered. Although we found the expected differences the 
variability along the variables that measure standard text was too large. 

2 ESD evoke differences in average reading time per word. Tliese differences are in line with 
our expectations. 

3 We have counted the number of jumps to the text: on average there are 3.6 jumps from a 
scheme to the rest of the text. 

Accepting these results, two out of three hypotheses can be rejected. 

We expected that when ESD are used at a deep level and not just read superficially, this could be 
observed in differences in eye movement patterns. The actual research has shown that deep level 
use of ESD is in fact measurable when considering reading time per word. 
In our opinion the statement that deep level use of ESD is measurable applies to ESD that elicit 
longer reading as well as to ESD that elicit shorter reading. Deep level use of an ESD can for 
instance imply that a student concentrates only on a certain part of the text and 'skips' the rest of 
the information. The latter is the case with 'examples' and 'content pages'. 

These results confirm the outcomes of previous research on the use of BSD. 

In Poelmans et al (1992) and Valcke et al (1993) we concluded that ESD are used at a deep level. 

Students do not only read them but they read them in function of their study process to reach a 
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better compreheasion. 

The research presented here also embodies an important learning* episode for our own future 
research practice. As far as we know the use of ESD was never before researched by means of eye 
movement registrations, although the technique of eye movement registration is commonly used 
(McConckie and Rayner, 1975; Rothkopf & Billington, 1979; Mandel, 1979; Just & Carpenter 1980; 
van Lieshout, 1982; Shebilske & Fisher 1981, 1983; Grabe et al, 1987; Blanchard & Iran-Nejad, 1987; 
Vauras, Hyona and Niemi, 1992; Britt et al, 1992). 

• In our opinion, the technique has proven to have the potential to deliver adequate and reliable 
results. For that purpose some minor and major adaptations of the research setup are required. 
These adaptations are described in the following section. 

6.2 Alternative set-up for experiments on eye movement registration and ESD 

Reconsideration of the experiments using eye movement registration for ESD-related research, 
shows that the early decision of using standard Ou course material as stimulus material for the 
experiments has made the final analysis and interpretation of the results very difficult. The limited 
resolution of the eye movement measurement technique, which is a direct result of the need for a 
low subject load, in combination with the small text font and huge information density of one page 
of Ou course material, makes that from ttie wide range of eye movement parameters (such as 
number of fixations and fixation length) only the reading time can be used. 
Reading time seems to be an important indicator for the distribution of attention on different areas. 
However, in the regions that are not dominated by text, such as schemes and pictures, additional 
parameters related to the identification of single fixations seem to be an important extra source of 
information. 

An additional problem resulting from the use of complete pages of very complex juridical course 
material is that there are numerous factors that may have an influence on reading time. It is hardly 
possible to quantify most of these factors. Boundary conditions for ecological validity set lower 
limits to the amount of the stimulus material in order to get a realistic educational setting. 
However, the actual analysis of eye movements should be limited to only part of the total amount 
of material containing a few well-chosen ESD's. Within this context it is important to verify whether 
the concept of standard text, in the sense of text wifliout ESD, is valid. This means that it is 
necessary to investigate to what extent 'eye movement related' characteristics for individual 
subjects can be defined for reading behaviour on flie type of study material that is used in the 
experiments. 

It is necessary to look for a presentation medium that allows for the presentation of only a limited 
amount of information at a time, which makes it possible to identify single fixations. On the other 
hand such an information packet must be part of a total package of study material with a realistic 
size. The presentation medium that is used must offer proper tools to the subjects and allow them 
navigate through the material in a proper way. 

If it is one of the aims of a follow-up research project to draw conclusions that are valid for groups 
of subjects, individual differences rnake it necesr ary to define a minimum number of subjects for 
the experiments. 

Summarizing we can say: 

- that the stimulus material that is to be used should be analyzed or even constructed property in 
the context of the research questions 

- more than it was the case in the current research, stimulus material should be analyzed and 
constructed at a micro level; the way in which ESD's affect eye movement parameters should be 
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isolated from other variables that may affect these parameters 

- when the stimulus material is constructed, the resolution of the eye movement equipment must 
be taken into account; a criterium can be that it must be possible to identify single words as target 
of single fixations. This would make it possible to use, besides total reading time per window and 
jumps from schemes, parameters such as number of fixations and fixation length. 



6.3 Future research 

With the adaptations described above eye movement registration can become a useful tool for 
future research on use of ESD. The promise for future research might especially be true if this 
objective measurement technique could be used in combination with the le^s objective research 
methods, such as questionnaire and interview. 

Due to the innovative character of our research, we devoted substantial attention to the potential 
distracting ii\fluence of the research set-up. We compared the study outcomes of the eleven 
students with the behaviour and characteristics of another sample that studied the same text but 
from a normal paper version. It was concluded that there is a small, but not significant, negative 
influence from the research set-up on study success. Also the interview sessions confirmed that the 
research set up was not very distracting. The latter might be due to the new and special 
elaboration of the eye movement registration device and the computer program ANALEYES for 
analysis which made possible that subjects had relatively large 'movement tolerance' (Stolk et al, 
1991). 

Originally, this research also was intended to investigate the influence of intra-individual student 
variables on the use of ESD: for instance "Do poor readers use ESD in a different way when 
compared to good readers? Are poor readers less well aware of functions of ESD and are they for 
instance less well able to distinct between basis information and examples in a study text?" Due to 
the limited amount of students involved, more advanced statistical analyses considering these 
intra-individual variables were irrelevant or impossible. Future research might be interesting. 

One disadvantage of the research method that remains is related to the high investment in time to 
gather and analyze the data. The research involves individual students and the analysis takes a lot 
of time. For the future we hope that it is possible to automatize - parts of - the research procedure. 
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